Guided inspection system and method

ABSTRACT

A system and method for a guided inspection of an apartment, home or other physical space is disclosed. The system and method use augmented reality to guide a user through a physical space. The system and method further use machine learning to automatically detect and classify damage to various physical structures in the physical space. In response to detected damage, the system may prompt a user to move closer to the detected damage for further inspection. The system can also detect obscured structures and prompt a user to make changes to the environment to increase the visibility of the obscured structures.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 16/288,629, filed Feb. 28, 2019 and titled “GuidedInspection System and Method”, which application claims the benefit ofU.S. Provisional Patent Application No. 62/721,501, filed Aug. 22, 2018,and titled “Guided Inspection System and Method,” the disclosures ofwhich applications are incorporated by reference herein in theirentirety.

TECHNICAL FIELD

The present disclosure generally relates to a guided inspection systemand method, and specifically to a system and method that use augmentedreality to guide a user through a physical space while automaticallydetecting possible damage.

BACKGROUND

Inspections are common for apartments and other rental properties.Before a new tenant moves in an inspection is performed (by the tenant,property manager, landlord or another party) to determine if there isany existing damage. During the inspection, the tenant makes note ofpossible issues or damage on a form that is submitted to the landlord,manager, or owner of the apartment/space. By noting pre-existing damagethe tenant can inform the landlord, manager or owner of issues thatpre-date the new tenant and therefore are not the financialresponsibility of the tenant. In many situations, tenants provide asecurity deposit prior to moving in. Any costs from damage to theproperty caused by the tenant may be subtracted from the securitydeposit, with any remainder being returned to the tenant at the end ofthe rental term. It is therefore in the financial interest of a would-betenant to make sure an inspection is performed with any pre-existingdamage noted in detail.

Inspections for rental properties may not be performed by professionals.In some cases, the inspection is filled out by the new tenant and/or thelandlord or manager. The inspection usually requires noting any damagedone to a long list of areas and structures provided on a form. The formcan be confusing and/or tedious to fill out.

There is a need in the art for a system and method that addresses theshortcomings discussed above.

SUMMARY

In one aspect, a method of guiding a user through an inspection of aphysical space includes steps of: (1) receiving image information from aremote device, the image information corresponding to at least one imageof the physical space; (2) analyzing the image information to determineif there is damage to a portion of the physical space based on the imageinformation from the remote device; (3) sending instructions to theremote device, the instructions including navigation information;receiving information that a location of the remote device has changed;and (4) sending new instructions to the remote device to capture newimages.

In another aspect, a method of guiding a user through an inspection of aphysical space using a remote device includes steps of: (1) capturingimages of the physical space; (2) sending image information to a server,the image information including at least one image of the physicalspace; (3) receiving instructions from the server, the instructionsincluding navigation information; (4) prompting the user to move to alocation in the physical space in response to the instructions; and (5)monitoring the movement of the remote device and confirming that theuser has moved to the location.

In another aspect, a method of guiding a user through an inspection of aphysical space includes steps of: (1) receiving image information from aremote device, the image information corresponding to at least one imageof the physical space; (2) detecting an obscured structure using theimage information; and (3) sending instructions to prompt the user tomodify the visibility of the obscured structure.

Other systems, methods, features, and advantages of the disclosure willbe, or will become, apparent to one of ordinary skill in the art uponexamination of the following figures and detailed description. It isintended that all such additional systems, methods, features, andadvantages be included within this description and this summary, bewithin the scope of the disclosure, and be protected by the followingclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention can be better understood with reference to the followingdrawings and description. The components in the figures are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the invention. Moreover, in the figures, likereference numerals designate corresponding parts throughout thedifferent views.

FIG. 1 is a schematic view of an embodiment of a user in an apartmentusing a remote device to perform a guided inspection;

FIG. 2 is a schematic view of an embodiment of a guided inspectionsystem;

FIG. 3 is a schematic view of a guided inspection process, according toan embodiment;

FIG. 4 is a schematic view of a guided inspection process performedusing a remote device and a server, according to an embodiment;

FIG. 5 is a schematic view of a process for assessing if there is damageto a physical space, according to an embodiment;

FIG. 6 is a schematic view of a situation where a system prompts a userto move closer to a region with possible damage, according to anembodiment;

FIG. 7 is a schematic view of a situation where a system prompts a userto confirm if there is damage in the highlighted region, according to anembodiment;

FIG. 8 is a schematic view of a process for determining if a structureis being obscured, according to an embodiment;

FIG. 9 is a schematic view of a situation where a window is obscured,according to an embodiment; and

FIG. 10 is a schematic view of a machine learning system, according toan embodiment.

DESCRIPTION OF EMBODIMENTS

The embodiments provide a system and method for guiding a user throughan inspection of a physical space, such as a rental apartment.Specifically, the embodiments provide a system and method that useaugmented reality to prompt a user to move to various locations in aphysical space, automatically analyze images of structures in thephysical space, and prompt the user further according to the results ofanalyzing the images. By automatically capturing and analyzing imageinformation about structures in the physical space to determine if thereis damage, the system and method improve the efficiency of theinspection process. By using an augmented reality system to prompt auser, the system and method simplify the inspection process and allowusers with little to no experience in inspecting properties to quicklyand accurately assess possible damage in a physical space.

As used herein, the terms “artificial intelligence” and “machinelearning” may be used to describe a variety of techniques in which analgorithm can learn to improve its performance on a task (for example,classifying images into different categories). The embodiments can makeuse of any known methods and systems in artificial intelligence and/ormachine learning.

As used herein, the term “augmented reality” refers to the ability tocombine computer generated sensory information (for example, images)with a real-world environment (for example, images or video of a room orother space). The embodiments also make use of methods and systemsrelated to the field of augmented reality. These include methods foridentifying and mapping features in a real-world environment, forgenerating images and/or other sensory data, and for augmenting thereal-world environment with the generated images/sensory data. Forexample, augmented reality systems (AR systems) may include thecapability to: sense a physical space using one or more cameras andbuild models of the space; generate virtual elements; and augment imagesof the physical space with the virtual elements using a display of somekind. Software tools for building AR systems are known and provided asopen source or commercial AR software development kits (SDKs).

An AR system may make use of various known methods, techniques, oralgorithms in robotics and/or navigation. For example, some embodimentsmay utilize the well known “simultaneous localization and mapping”(SLAM) technique for constructing and updating a map of an unknownenvironment and determining the location (and pose) of an agent withinthe map. Some implementations of SLAM can be used to help identifyobjects, determine distances between objects, determine the dimensionsof objects, position objects in a virtual space and/or performtransformations of virtual objects (such as rotation). Various kinds ofSLAM techniques are known and adapted to particular kinds of tasks.These include EKF SLAM, FastSLAM, Graph-based SLAM, Topological SLAM andVisual SLAM.

FIG. 1 is a schematic view of a user 100 in a room of an apartment. Inthe present embodiment, user 100 has a phone 104 that is running asoftware application 106. Application 106 provides the user interfacefor an augmented reality guided inspection system that utilizes bothaugment reality and machine learning. The guided inspection system,described in further detail in FIG. 2 , includes a variety of featuresthat may be used to guide a user through an apartment, house or otherphysical space to perform an automated, or semi-automated, inspection.The system could be used during an apartment inspection that is oftenperformed by a tenant prior to moving into an apartment.

FIG. 2 is a schematic view of an embodiment of a guided inspectionsystem 200, also referred to simply as system 200. System 200 mayinclude various sub-systems and other components that facilitate guidinga user through a physical space and automatically performing one or morekinds of inspection tasks. In some cases, the process of guiding a usermay be facilitated by augmented reality, including visual displays tohelp the user navigate through a physical space. In some cases, theprocess of automatically performing one or more kinds of inspectiontasks can include using machine learning to detect and classify possibledamage.

Guidance system 200 may comprise a centralized computer system 202 and aremote device 204 that may communicate with one another through anetwork 206. The term “computer system” refers to the computingresources of a single computer, the partial computing resources of asingle computer, a plurality of computers communicating with oneanother, or a network of remote servers. In an exemplary embodiment,computer system 202 includes at least one server.

Centralized computer system 202 may receive various kinds of informationfrom remote device 204 (or other sources), perform various kinds ofanalyses and/or store data. Whereas centralized computer system 202 maybe located anywhere, remote device 204 may be located on site (forexample, with a user at an apartment building) to facilitate thecollection of data for an inspection.

In the embodiment of FIG. 2 , centralized computer system 202 comprisesone or more computing devices 210 (for example, a server) that may be incommunication with one or more databases 212. Databases 212 could beco-located with computing device 210 or could be remote databases thatare accessible by computing device 210 over network 206. Databases 212can include any kind of storage devices, including but not limitedmagnetic, optical, magneto-optical, and/or memory, including volatilememory and non-volatile memory.

Remote device 204 may comprise a device that can be brought to thelocation where an inspection is to occur. Remote device 204 can comprisea computer system for processing and communicating information. A remotedevice may generally include a processor, a data storage component and adisplay. A remote device may also include components to facilitatecommunication with external systems (for example, hardware and softwarecomponents to enable communication over network 206). In some cases, aremote device includes one or more physical buttons. In some cases, aremote device includes touchscreen controls. Still further, remotedevice 204 can include speakers and a microphone for receiving andgenerating audible sounds. In the exemplary embodiment of FIG. 2 ,remote device 204 comprises a tablet computing device. In otherembodiments, however, a remote device could comprise a smartphone, alaptop, or similar kind of device.

Remote device 204 may include hardware components for capturing sensoryinformation, as well as storing and/or transmitting capturedinformation. As used herein the term “sensory information” can includevisual information, audible information, tactile information and/orinformation related to the motion of the remote device (for example,acceleration information). In an exemplary embodiment, remote device 204includes a camera for capturing images in the form of photos or video.Remote device 204 may also include an accelerometer and/or gyroscope fordetecting linear accelerations and/or angular rotational velocity. Insome cases, accelerometer and/or gyroscope data can be used by an ARsystem to build a map of a physical space and locate the remote devicewithin the map.

Remote device 204 may include additional sensors including, but notlimited to: a proximity sensor to detect proximity to one or moreobjects in a physical space, an ambient light sensor for detectingambient light conditions in a physical space, and a compass fordetecting directional information. Additionally, in some embodiments,remote device 204 may include a GPS receiver for receiving GPSinformation that can be used to determine the location of the remotedevice.

Remote device 204 may run one or more software applications thatfacilitate guiding a user through a physical space for purposes ofinspection. These applications could be native to the device's operatingsystem or web-applications that run on a browser. Moreover, anapplication may be configured with a graphical user interface (GUI) thatfacilitates visual and/or tactile interaction between a user andelements of a guided inspection system. As one example, FIG. 1 depicts anative software application 106 running on phone 104 that is used toguide a user through an apartment to perform an inspection.

A remote device and a centralized computer system could operate in aclient-server relationship. For example, centralized computer system 202may include a server that communicates with remote device 204 as well asother remote devices. In some cases, multiple remote devices runningindividual instances of a guidance inspection application could operateas clients in communication with centralized computer system 202 overnetwork 206.

FIG. 3 is a schematic view of a process of guiding a user through aphysical space of some kind with a guided inspection system. Thephysical space could be an apartment, a condominium, a single-familyhome, a business property or some other physical space. Moreover, asused herein, a physical space could refer to only a portion of anapartment, single family home, or other physical space. The followingsteps may be performed by one or more components of a guided inspectionsystem.

In a first step 302, a user may be guided through a living space (i.e.,a physical space). Guidance can be accomplished using a remote device(for example, remote device 204). A remote device can direct a userthrough a living space via one or more guidance prompts. The term“guidance prompt” could refer to displayed text, audible speech, othervisual indicators and/or some combination of these. In one embodiment, asystem could display a map of the physical space and display a path (orarrow) from the user's current location to another location where thesystem wants to user to move. In another embodiment, a system couldprovide verbal commands, such as “move forward 3 feet”, “move closer tothe wall in front of you”, or “move to the kitchen”. Such verbalcommands could be spoken or displayed as text on the remote device.

Generally, a guided inspection system may guide a user to locations in aphysical space where a physical structure can be examined. As usedherein, the term “physical structure” refers to some identifiablefeature in a physical space. Examples of physical structures include,but are not limited to: walls, doors, door frames, windows, blinds,curtains, ceilings, floors, carpets, counters, cabinets, tables, lightfixtures, electrical sockets, appliances, vents, toilets, bathtubs,sinks, as well as other structures. Physical structures could includeboth fixed and moveable structures. As used herein, the term “fixedstructure” refers to a structure with a fixed position within thephysical space, which cannot be displaced without disassembly ordestruction. Examples of fixed structures include walls, doors andcabinets. Examples of moveable structures include furniture andappliances. In some property inspections the focus is primarily on fixedstructures and some appliances.

It is contemplated that an inspection may proceed in a predeterminedmanner or in an ad hoc manner. In a predetermined inspection, a guidedinspection system is aware of a set of target physical structures to beinspected. For example, a guided inspection system could be providedwith a map of the physical space, noting the existence and relativelocations of various physical structures such as walls, ceilings,floors, doors, rooms, bathrooms, as well as other physical structures.In such a situation the system could guide a user from feature tofeature in a predetermined sequence. Alternatively, in an ad hocinspection, a guided inspection system may not be aware of one or morephysical structures ahead of time. In such a situation the system coulddirect a user to move from one physical structure to another in a moread hoc manner. In some cases, the system could simply ask a user toproceed to another physical structure without knowing about the feature.For example, the user could be prompted to “move to the next room”, orto “move to another door in the apartment”, or to “move to the door ofthe hallway bathroom”. In some embodiments, a system could learn newinformation about a physical space in real time and use that informationto direct the user to another physical structure that it has recentlyidentified.

In step 304, a guided inspection system may instruct a user to captureimages or other sensory information about particular physicalstructures. For example, the system could prompt a user to “focus cameraon nearby doorknob and take a picture.” Alternatively, the system couldautomatically take pictures (or video) of one or more physicalstructures as they are automatically identified by an imagedetection/recognition algorithm. In some cases, a user may be promptedto aim the camera at a particular physical structure and the system mayautomatically capture images.

In some embodiments, a user may only be instructed to capture an imageof a physical structure if there is visible damage. In otherembodiments, a user may be instructed to capture images of variousphysical structures and the system may automatically detect and classifydamage without any user input.

Next, in step 306, a guided inspection system can automaticallyidentify/classify and catalog damage to physical structures. Forexample, a system could automatically detect that a living room carpetis damaged based on images of the carpet. The system may also classifythe type of damage (for example, as a tear in the rug). Afteridentifying and classifying the damage, the system may catalog (orstore) relevant data in the form of damage information. As used herein,the term “damage information” refers to any information corresponding tosome aspect of damage to a physical structure. Damage information couldinclude the type of feature (for example, a wall or a floor) and thetype of damage (for example, a tear or a crack).

The catalog of damage information can be used to build an inspectionreport that lists physical structures in the physical space and theexistence and/or type of damage to each physical structure.Alternatively, rather than generating an inspection report, the damageinformation could be stored in some kind of data structure for laterretrieval and use.

Optionally, in at least some embodiments, system 200 may captureinformation about various objects in a physical space and create aninventory of the objects that can be used to determine quotes forfinancial products such as renter's or home owner's insurance. In somecases, the objects that are captured and identified may be objects ownedby the tenant (or owner), such as televisions and other electronics,jewelry and artwork, and clothing. These objects may be distinct fromthe fixed physical structures that form part of the property itself,such as the walls, doors and other structures that are often inspectedprior to leasing an apartment or other property. Methods ofautomatically collecting image information about objects in a home,apartment or other property and automatically identifying andcataloguing information about those objects are disclosed in U.S. Pat.No. 11,055,531 titled “AUGMENTED REALITY METHOD FOR REPAIRING DAMAGE ORREPLACING PHYSICAL OBJECTS”, the entirety of which is hereinincorporated by reference and referred to as the “Method of RepairingDamage” application hereafter. The embodiments can make use of any ofthe systems and methods described in the Method of Repairing Damageapplication, thereby allowing a guided inspection system to capture andanalyze images of objects and generate estimated quotes for renter'sinsurance.

The exemplary process depicted in FIG. 4 shows that some steps could beperformed by a remote device (for example, remote device 204) and othersteps by a component of a centralized computer system (for example, aserver 203 of centralized computer system 202). In other embodiments,some steps shown as performed by a remote device could be performed by acentralized computer system or vice-versa.

Initially, a system may guide a user through a living space (i.e., aphysical space), as in step 302 of FIG. 3 . In some embodiments, remotedevice 204 may prompt a user to move to a particular location in aphysical space, such as a bedroom. As described above, this prompting ofthe user could be achieved using displayed text, audible speech and/orother displayed visual information such as a map with a route.

As the user is guided through the living space, remote device 204 maycapture images (photos or video) of a living space (i.e., a physicalspace) during step 402. In some cases, remote device 204 may prompt auser to aim the camera and/or take images of one or more physicalstructures at the location. For example, if the user is in a bedroomremote device 204 may prompt the user to take images of the bedroomwindow to check for damage. In some other embodiments, remote device 204may automatically take pictures or video without prompting a user.Optionally, remote device 204 could prompt a user to aim the camera at acertain area or feature in the room but may take images or videosautomatically without further user action.

In step 404, remote device 204 sends image information to server 203 ofcentralized computer system 202 over a network (for example, network206). The term “image information”, as used herein, refers to anyinformation corresponding to photos or videos. Image information couldbe stored in any known image file formats, such as JPEG, TIFF, GIF, PNGand BMP. Image information could also be stored in any known video fileformats, such as AVI, FLV, WMV, MOV and MP4.

Next, in step 406, server 203 may receive the image information fromremote device 202. Following this, in step 408, server 203 may analyzethe image information and identify potential damage. Specifically, theimage information may be processed by one or more machine learningand/or machine vision algorithms to detect and classify physical damageto one or more physical structures. As an example, during step 408,server 203 could receive an image of a window. This image could be inputinto a machine learning module configured to detect and classify damage.The output of the module may be damage information, such as whether ornot damage was detected (yes or no) and the classification of anydetected damage (for example, broken window). This damage informationmay be stored locally, stored on a database (for example, databases 212)and/or sent to remote device 204.

In step 410, server 203 may prepare and send instructions (and/or otherinformation) to remote device 204. The instructions may be based oninformation from step 408. For example, after server 203 determines thestate (i.e., damaged or undamaged) of a bedroom window from an imageduring step 408, server 203 could prepare and send instructions toremote device 204 to prompt the user to move on to another room or adifferent feature in the room.

Based on the instructions that remote device 204 receives during step412, remote device 204 may prompt the user to move to a new location instep 414. In step 416, remote device 204 may confirm that the user hasmoved to the new location. The remote device may confirm that the userhas moved using various methods and available systems. For example, theremote device could monitor accelerometer information from anaccelerometer (and/or gyroscopic information from a gyroscope sensor) todetermine that the user has moved the desired direction and/or distance.In some cases, the remote device could use image information todetermine if/where a user has moved. In some cases, an augmented realitysystem may use image information from a camera along with accelerationand angular velocity information to determine if the user has moved tothe requested location. As another example, the remote device could useGPS information to confirm that the user has moved to the requestedlocation. At this point remote device 204 could continue prompting theuser to capture new image information for processing by server 203.

The embodiment of FIG. 4 is characterized by a process where steps 406,408 and 410 are performed by a server of a centralized computer system.Optionally, in some other embodiments, some of steps 406-410 couldperformed by remote device 204.

Referring next to FIG. 5 , in some embodiments the system may undergo afeedback loop when attempting to identify and classify damaged physicalstructures. In one embodiment, the steps illustrated in FIG. 5 may beperformed by centralized computer system 202 as part of the process foranalyzing image data and determining if there is damage.

At step 502, centralized computer system 202 may analyze imageinformation corresponding to a physical structure in a physical space.For example, image information corresponding to a particular physicalstructure may be provided as input to a machine learning system thatdetects and/or classifies physical damage. Based on the output of themachine learning system, centralized computer system 202 may decide whataction to take at step 504. If there is damage detected, centralizedcomputer system may move to step 506 to store damage information relatedto the structure. Following this centralized computer system 202 mayproceed to step 508 to select the next structure to analyze, and maythen return to step 502 to analyze images of the new structure. If,during step 504 centralized computer system 202 determines that there isno damage, centralized computer system 202 may proceed immediately tostep 508 to select the next structure to analyze.

It may be the case that a machine learning system outputs a result thatindicates substantial uncertainty in detecting damage to a structure.For example, the machine learning system could be designed to output arange of probabilities that a structure is damaged. Outputs with arelatively high probability of damage (for example, 80-100%) are assumedto have damage and outputs with a relatively low probability of damage(for example, 0-20%) are assumed to have no damage. Outputs with anintermediate probability (for example, between 20 and 80%) may betreated as too uncertain to label as damaged or undamaged. In this case,the guided inspection system could take actions to gather additionaldata to achieve a more accurate detection/classification.

If, during step 504, centralized computer system 202 determines there isuncertainty in classifying a structure (or an image corresponding to thestructure) as damaged or not damaged, centralized computer system 202may proceed to step 510. In step 510, centralized computer system 202may prepare and send instructions to guide the user closer to thestructure to obtain additional image information. Specifically, in somecases, the instructions are submitted to remote device 204.

Next, in step 512, centralized computer system 202 may wait for newimage information. Once the new image information has been receivedcentralized computer system 202 proceeds back to step 502 to analyze thenew image information. Such a process may be iterated until centralizedcomputer system 202 has sufficiently good image information to determinewith high confidence if the structure is or isn't damaged.

In some embodiments, the instructions sent from centralized computersystem 202 to remote device 204 may include navigation information. Asused herein, the term “navigation information” refers to any informationthat can be used in determining a location and/or providing directionsor instructions from one location to another. In the context ofnavigating physical spaces such as apartments, houses and otherbuildings, navigation information can include relative and absolutelocation information. Absolute location information can include rooms,GPS coordinates or other geographical information. Absolute locationinformation may also be identified with the locations of known fixedstructures such as doors, walls, windows, countertops and support beams.An example of relative location information includes giving a lineardistance from a known fixed structure (such as a set distance from aparticular window).

To guide a user through a physical space and provide instructions tomove relative to a structure, or to move between different structures, aguided inspection system may incorporate a model of the physical space.In some embodiments, the model may be implemented by an augmentedreality system that identifies new structures (for example, walls anddoors) and builds a real-time model of both the physical space and theremote device/camera within the physical space. Using this model, thesystem can provide navigation instructions to a user. Specifically, thesystem can provide navigation instructions directing the user to movecloser to a structure, or more generally to move between two differentlocations within the physical space.

To build a real-time model of the physical space and locate and trackthe changing position and/or orientation of the remote device in thephysical space, a system may make use of multiple modes of sensory datafrom the remote device. For example, some embodiments may use acombination of image information from the camera of a remote device,acceleration information from an accelerometer of the remote device andangular velocity information from a gyroscope of the remote device tomap out the physical space in real time and orient/track the remotedevice through the physical space. As described above, some AR softwarealgorithms may use SLAM type algorithms (such as Visual SLAM) to buildthe model of the physical space and track the remote device (and user)through the space.

Apart from using augmented reality techniques, a system could makedirect use of GPS information and/or map information provided ahead oftime. In some cases, maps of the interior of a rental structure may beavailable from a third party and/or from the owner/manager of the rentalproperty. A system could retrieve a map of the physical space and useGPS information received through a GPS receiver to track movement andposition of the remote device/user.

FIGS. 6 and 7 depict an exemplary situation where a user is guidedcloser to a wall based on uncertainty in identifying damage to the wall.In FIG. 6 , a user is holding remote device 204 in the direction of wall602. Remote device 204 has captured images of a wall 602 in a bedroom604. After processing the image information corresponding to thecaptured images, the guided inspection system is unable to determine ifthe image should be classified as showing damage or not. Because of thisuncertainty, centralized computer system 202 sends back instructions toremote device 204. Based on these instructions, remote device 204prompts the user to move to a location that is closer to a possiblecrack in the wall. In the exemplary embodiment, remote device 204 mayuse augmented reality to display a path 608 and a destination over thereal-time image 610 of the physical space.

In FIG. 7 , the user has moved closer to the door resulting in remotedevice 204 obtaining better quality images of the region just above door620. At this point the newly obtained image information is sent tocentralized computer system 202 for processing and assessment.Centralized computer system 202 returns damage information and remotedevice 204 alerts the user of damage that has been detected. In theexample of FIG. 7 , the system may prompt the user to confirm that thereis in fact damage to one or more structures shown on the screen. Toclarify what structure is possibly damaged, remote device 204 maydisplay a highlighted boundary 720 around the damage in the live videofeed of the area.

While the embodiment of FIG. 7 depicts text-based prompts to provideinstructions to a user, audible prompts, in the form of computergenerated speech and/or other kinds of prompts could also be used. Insome cases, multiple types of prompts can be used simultaneously,including a combination of text/indicia and spoken instructions.

It may be appreciated that during this process centralized computersystem 202 could provide other kinds of instructions. As one example, ifan image processed at centralized computer system 202 is out of focus ornot centered sufficiently on a given physical structure, centralizedcomputer system 202 may prepare and send instructions to have imagesretaken, either manually by a user or automatically by remote device204.

A guided inspection system may also include provisions for detectingwhen a structure to be inspected is obscured. For example, during aninspection the blinds on a window may be down. This allows the system todetermine if the blinds are damaged, but obscures the window itself fromview. In that situation a guided inspection system could be configuredto automatically detect the obscured window and prompt the user to raisethe blinds so the window can be inspected.

A process for automatically depicting obscured objects is shown in FIG.8 . In step 802, centralized computer system 202 may analyze imageinformation corresponding to a particular structure. Next, in step 804,centralized computer system 202 may determine if any structuresassociated with the image information are being obscured. If not,centralized computer system 202 proceeds to step 806 to select anotherstructure to analyze. In some cases, the system can then proceed back tostep 802. If it is determined during step 804 that there is an obscuredstructure, centralized computer system 202 proceeds to step 808.

In step 808, centralized computer system 202 sends instructions to aremote device. The instructions indicate that there is an obscuredstructure and action must be taken to make the obscured structurevisible. The instructions may also include a request for the user and/orremote device to take new images once the structure is visible. Forexample, if the blinds are down centralized computer system 202 may sendinstructions to raise the blinds and take new images of the window. Anexample in which remote device 204 has received instructions to prompt auser about an obscured window 902 is depicted in FIG. 9 . In this case,remote device 204 prompts the user using text based instructions 904 aswell as a visual indicator 906 displayed by an AR system.

In step 810, centralized computer system 202 waits for updatedinformation from a remote device. Once the new image information isreceived, centralized computer system 202 proceeds back to step 802 toanalyze the image information. This process can be iterated until eachobscured structure is properly imaged.

While FIG. 9 provides one example of an obscured window andcorresponding instructions for making the window visible, it may beappreciated that a guided inspection system can be configured to detectand provide instructions for various other situations where a structureis obscured. In another situation, the blinds on a window may be raised.The system may automatically detect that the blinds are raised andprompt the user to lower them so they can be inspected.

In some situations, one structure may be obscured by another object,such as a piece of furniture. In such a situation, a guided inspectionsystem can automatically detect both the object and obscured structureand prompt a user to move to a new location where the structure ofinterest won't be obscured.

In another example, a guided inspection system may be configured todetect ambient light conditions. This may be done using a built inambient light sensor on a remote device, for example. If the systemdetermines that the lighting conditions are poor for taking images inall or part of a physical space, it may automatically turn on a cameralight on the remote device (or instruct a user to do so).

Embodiments can use any combination of processes described above andshown in FIGS. 3-9 . For example, in one embodiment, a centralizedcomputer system may combine processes for dealing with uncertainclassifications as shown in FIG. 6 and processes for dealing withobscured structures as shown in FIG. 8 . That is, when a system isanalyzing image information it may detect both potential damage andobscured structures at the same time.

A guided inspection system may provide various kinds of outputs duringand/or after an inspection has been completed. In some embodiments, thesystem can generate an automated report or inspection form. For example,following an inspection a user could have the system generate aninspection form listing any damages. The user could then have the systemsend that inspection form to the landlord, property manager, owner orother party.

In some embodiments, a guided inspection system could be used to helpwith routine maintenance. For example, during the guided inspection thesystem could have the user inspect the batteries in any smoke detectorsthat it automatically identifies. The system could also be configured toidentify other parts that may need regular maintenance, like hoses,pipes, wires, filters or other replaceable structures. Any softwarerunning on the remote device could be configured to provide regularmaintenance reminders to a user. After being reminded, the user couldactivate a new instance of the guided inspection in a “maintenance”mode, in which the system primarily focusses on ongoing maintenanceissues.

In some embodiments, images, damage information and/or other kinds ofinformation gathered or generated during an inspection could be storedfor later use by the system. In one embodiment, the data for a givenphysical space could be stored as part of an ongoing record. Thisongoing record could provide a historical record of the apartment,house, or other property that could be provided to future tenants,owners, managers or other parties.

To detect and classify structures and/or damage, the embodiments mayutilize a machine learning system. As used herein, the term “machinelearning system” refers to any collection of one or more machinelearning algorithms. Some machine learning systems may incorporatevarious different kinds of algorithms, as different tasks may requiredifferent types of machine learning algorithms. Generally, a machinelearning system will take input data and output one or more kinds ofpredicted values. The input data could take any form including imagedata, text data, audio data or various other kinds of data. The outputpredicted values could be numbers taking on discrete or continuousvalues. The predicted values could also be discrete classes (forexample, a “damaged” class and an “undamaged” class). Numerical outputscould represent a probability that the input belongs to a variousclasses. Moreover, it may be appreciated that the same machine learningsystem can be used for training, testing and deployment, in some cases.

Referring to FIG. 10 , a guided inspection system may use machinelearning system 1000 to detect and classify structures and/or damage tostructures. In the example of FIG. 10 , machine learning system 1000 istrained on input data 1002. Input data 1002 comprises various images ofdamaged structures (cracked wall 1010, broken blinds 1012, shatteredwindow 1014 and torn carpet 1016). Although not shown in FIG. 10 , thetraining data may also include images of undamaged structures so thatthe system can learn to distinguish between damaged structures andundamaged structures. In some cases, machine learning system 1000 maycontinue to train on new data as it is received from ongoing use of theguided inspection system by many different users. Furthermore, becausethe system can optionally ask users to confirm if images have beencorrectly identified/classified, this provides a means of generating newsupervised training data with the image and known classification beingprovided by users of the system.

In order to learn to detect obscured objects, machine learning system1000 may also be trained on images of obscured objects. These mayinclude images of closed blinds (obscuring the window) and open blinds(obscuring the blinds). These may also include images of doors open to aposition that obscures a wall or other structure behind the door. Thus,the machine learning system can be trained to detect both damagedstructures and obscured structures or objects.

The output of machine learning system 1000 is a set of classes. Forsimplicity, only four possible classes are depicted here, correspondingto broken window 1020, torn carpet 1022, crack 1024 and broken blinds1026. Of course, the system could be configured with a large number ofpossible classes. Though FIG. 10 depicts a set of possible outputs(classes), for a given input image the system may generally predict asingle class from the set of possible classes. Optionally, the systemcould be configured to select two or more classes where there may beoverlap. For example, an image of a room with a large crack runningalong the wall and ceiling could trigger the system to output theclasses “crack in wall” and “crack in ceiling”.

Techniques from artificial intelligence and machine learning could beused for image detection and/or recognition as well as for otherpurposes. For example, the embodiments could make use of any methodsfrom the field of machine (or computer) vision including methods,techniques, or algorithms in machine vision and/or feature detection toidentify and classify objects. Embodiments may use any known imageprocessing methods such as stitching/registration, filtering,thresholding, pixel counting, segmentation, edge detection, coloranalysis, blob detection, pattern recognition or template matching,optical character recognition as well as other known methods. Someembodiments may use the scale-invariant feature transform (SIFT)algorithm that is used in object recognition, robotic mapping and imagestitching. Embodiments may also use known techniques in deep learning tohelp process and classify objects within image data. These techniquesinclude various kinds of deep neural networks. In some cases,embodiments may use one or more kinds of convolutional deep neuralnetworks (CNNs) that are commonly used in image recognition and otherareas of machine vision.

In some embodiments, various systems such as an AR system and/or amachine learning system could be implemented on a centralized computersystem. In some embodiments, an AR system and/or a machine learningsystem could be provided through a cloud service. In still otherembodiments, an AR system and/or a machine learning system could beintegrated into software running on a remote device. Moreover, in someembodiments, some components or software modules of a system could runlocally on a remote device while other components or modules run on acentralized computer system. For example, an AR system could havemodules running on a remote device for storing model parameters andinterfacing with sensors, and user interaction components (screen,controls, etc.). The AR system could also have modules running on acentralized computer system for more intensive processing tasks.Likewise, a machine learning system could be configured with somemodules running directly on the remote device and other modules runningon a centralized computer system for more intensive processing tasks.

It may be appreciated that given sufficient processing power and memory,some or all components of an AR system and/or a machine learning systemcould be implemented on a remote device (such as a tablet computer). Insuch embodiments, tasks described above as being completed by acentralized computer system or server could be handled by softwaremodules implemented on the remote device. As one example, though manymachine learning algorithms require intensive processing for training,once the parameters of a machine learning model (such as a neuralnetwork) have been learned the deployed machine learning algorithm orsystem may be less computationally intensive to run and could beconfigured to run efficiently on a mobile computing device such as asmart phone or tablet computer.

The processes and methods of the embodiments described in this detaileddescription and shown in the figures can be implemented using any kindof computing system having one or more central processing units (CPUs)and/or graphics processing units (GPUs). The processes and methods ofthe embodiments could also be implemented using special purposecircuitry such as an application specific integrated circuit (ASIC). Theprocesses and methods of the embodiments may also be implemented oncomputing systems including read only memory (ROM) and/or random accessmemory (RAM), which may be connected to one or more processing units.Examples of computing systems and devices include, but are not limitedto: servers, cellular phones, smart phones, tablet computers, notebookcomputers, e-book readers, laptop or desktop computers, all-in-onecomputers, as well as various kinds of digital media players.

The processes and methods of the embodiments can be stored asinstructions and/or data on non-transitory computer-readable media.Examples of media that can be used for storage include erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memories (EEPROM), solid state drives, magneticdisks or tapes, optical disks, CD ROM disks and DVD-ROM disks.

The embodiments may utilize any kind of network for communicationbetween separate computing systems. A network can comprise anycombination of local area networks (LANs) and/or wide area networks(WANs), using both wired and wireless communication systems. A networkmay use various known communications technologies and/or protocols.Communication technologies can include, but are not limited to:Ethernet, 802.11, worldwide interoperability for microwave access(WiMAX), mobile broadband (such as CDMA, and LTE), digital subscriberline (DSL), cable internet access, satellite broadband, wireless ISP,fiber optic internet, as well as other wired and wireless technologies.Networking protocols used on a network may include transmission controlprotocol/Internet protocol (TCP/IP), multiprotocol label switching(MPLS), User Datagram Protocol (UDP), hypertext transport protocol(HTTP) and file transfer protocol (FTP) as well as other protocols.

Data exchanged over a network may be represented using technologiesand/or formats including hypertext markup language (HTML), extensiblemarkup language (XML), Atom, JavaScript Object Notation (JSON), YAML, aswell as other data exchange formats. In addition, informationtransferred over a network can be encrypted using conventionalencryption technologies such as secure sockets layer (SSL), transportlayer security (TLS), and Internet Protocol security (Ipsec).

While various embodiments of the invention have been described, thedescription is intended to be exemplary, rather than limiting, and itwill be apparent to those of ordinary skill in the art that many moreembodiments and implementations are possible that are within the scopeof the invention. Accordingly, the invention is not to be restrictedexcept in light of the attached claims and their equivalents. Also,various modifications and changes may be made within the scope of theattached claims.

The invention claimed is:
 1. A method of guiding a user through aproperty inspection of an apartment using a remote device, comprisingthe steps of: capturing image information with the remote device, theimage information corresponding to at least one image of the apartment,wherein the at least one image includes an image of a physical structureat a first location inside the apartment; analyzing the imageinformation to determine if there is damage to a portion of the physicalstructure in the apartment based on the image information captured withthe remote device, wherein analyzing the image information comprisesusing a machine learning system to detect damage to the physicalstructure and the method further includes training the machine learningsystem using a set of images showing damage to physical structures;determining, by the machine learning system, that there is uncertaintyin classifying the physical structure at the first location as damagedor undamaged based on the image information for the physical structurecaptured with the remote device at the first location; based ondetermining by the machine learning system that there is uncertainty inclassifying the physical structure as damaged or undamaged, instructingthe user of the remote device to move closer to the physical structureand to capture new image information of the physical structure from alocation closer to the physical structure than the first location; uponclassifying the physical structure at the first location as damaged orundamaged by the machine learning system, providing navigationinformation on a display of the remote device to guide the user to asecond location inside the apartment; determining that a location of theremote device has changed from the first location to the secondlocation; and instructing the user of the remote device to capture newimages of the apartment at the second location.
 2. The method accordingto claim 1, wherein the property inspection includes a predetermined setof target physical structures inside the apartment to be inspected; andwherein providing the navigation information includes instructions forthe user to move to another location in the apartment corresponding toat least one target physical structure in the set of target physicalstructures.
 3. The method according to claim 1, wherein the machinelearning system is implemented by at least one processor of acentralized computer system or a cloud service.
 4. The method accordingto claim 1, wherein the machine learning system is implemented by aprocessor of the remote device.
 5. The method according to claim 1,wherein analyzing the image information by the machine learning systemto detect damage to the physical structure comprises outputting aprobability that the physical structure is damaged.
 6. The methodaccording to claim 5, wherein, when the probability is within a firstrange of probabilities, damage to the physical structure is determined;wherein, when the probability is within a second range of probabilitiesthat is less than the first range of probabilities, the physicalstructure is determined to be undamaged; and wherein, when theprobability is within a third range of probabilities that is less thanthe first range of probabilities and more than the second range ofprobabilities, damage to the physical structure is determined to beuncertain.
 7. A remote device for guiding a user through a propertyinspection comprising: a camera; a display; and at least one processorconfigured to implement instructions stored in memory of the remotedevice to cause the remote device to: display a map of an interior of ahome on the display of the remote device, the map including a pluralityof physical structures located within the interior of the home alongwith an associated location for each physical structure of the pluralityof physical structures; display a path on the map on the display of theremote device from a current location of the remote device to a firstlocation associated with a first physical structure of the plurality ofphysical structures; capture image information of the first physicalstructure inside the home at the first location; send the imageinformation to a server, the image information including at least oneimage of the first physical structure inside the home; receiveinstructions from the server, the instructions including navigationinformation directing the user from the first location to a secondlocation inside the home, the second location being associated with asecond physical structure of the plurality of physical structures insidethe home, the second physical structure being different from the firstphysical structure; prompt the user on the remote device to move to thesecond location in the home in response to the instructions, whereinprompting the user includes displaying a path on the map on the displayof the remote device from the first location to the second location;monitor movement of the remote device to confirm that the user has movedto the second location; and upon confirming that the user has moved tothe second location, prompt the user on the remote device to capture animage of the second physical structure at the second location.
 8. Theremote device according to claim 7, further comprising an accelerometer;and wherein the instructions implemented by the processor to cause theremote device to monitor the movement of the remote device includereceiving information from the accelerometer.
 9. The remote deviceaccording to claim 7, wherein the map of the interior of the apartmentincludes a predetermined sequence of the plurality of physicalstructures that are to be inspected.
 10. The remote device according toclaim 7, wherein the instructions implemented by the processor to causethe remote device to direct the user with the navigation informationfurther include displaying directions to the second location on thedisplay of the remote device using augmented reality.
 11. The remotedevice according to claim 7, wherein the instructions implemented by theprocessor for prompting the user cause the remote device to direct theuser to move the remote device closer to the physical structure.
 12. Theremote device according to claim 7, wherein the instructions implementedby the processor further cause the remote device to indicate a physicalstructure with potential damage on the display of the remote deviceusing augmented reality.
 13. The remote device according to claim 12,wherein the instructions implemented by the processor further cause theremote device to prompt the user to confirm that the indicated physicalstructure is damaged.
 14. The remote device according to claim 7,wherein prompting the user includes displaying text on the display. 15.The remote device according to claim 7, further comprising speakers; andwherein the instructions implemented by the processor to cause theremote device to prompt the user include generating spoken words to beplayed through the speakers.
 16. A remote device for guiding a userthrough a property inspection of an apartment comprising: a camera; adisplay; and at least one processor configured to implement instructionsstored in memory of the remote device to cause the remote device to:capture image information using the camera, the image informationcorresponding to at least one image of the apartment; detect an obscuredphysical structure inside the apartment using the image information,wherein detecting the obscured physical structure includes inputting theimage information into a machine learning system that has beenpreviously trained using a set of images showing obscured physicalstructures; provide instructions to the user on the remote device tomodify the visibility of the obscured physical structure so that thephysical structure is no longer obscured, wherein the instructionsprompt the user to change the state of a window treatment to an open orclosed position so that the physical structure is no longer obscured inorder to capture an image of the physical structure; capture new imagesof the physical structure using the camera that are no longer obscured;and based on the new images of the physical structure that are no longerobscured, determine, by the machine learning system, whether thephysical structure is damaged or undamaged.
 17. The remote deviceaccording to claim 16, wherein the instructions implemented by theprocessor further cause the remote device to prompt the user to changethe lighting conditions in the apartment.
 18. The remote deviceaccording to claim 16, wherein determining, by the machine learningsystem whether the physical structure is damaged or undamaged comprisesoutputting a probability that the physical structure is damaged.
 19. Theremote device according to claim 18, wherein, when the probability iswithin a first range of probabilities, damage to the physical structureis determined; wherein, when the probability is within a second range ofprobabilities that is less than the first range of probabilities, thephysical structure is determined to be undamaged; and wherein, when theprobability is within a third range of probabilities that is less thanthe first range of probabilities and more than the second range ofprobabilities, damage to the physical structure is determined to beuncertain.
 20. The remote device according to claim 19, wherein themachine learning system is trained using a set of images showing damageto physical structures and another set of images showing undamagedphysical structures.